Disentangling Bias and Variance in Election Polls

It is well known among researchers and practitioners that election polls su↵er from a variety of sampling and non-sampling errors, often collectively referred to as total survey error. Reported margins of error typically only capture sampling variability, and in particular, generally ignore non-sampling errors in defining the target population (e.g., errors due to uncertainty in who will vote). Here we empirically analyze 4,221 polls for 608 state-level presidential, senatorial, and gubernatorial elections between 1998 and 2014, all of which were conducted during the final three weeks of the campaigns. Comparing to the actual election outcomes, we find that average survey error as measured by root mean square error (RMSE) is approximately 3.5 percentage points, about twice as large as that implied by most reported margins of error. We decompose survey error into election-level bias and variance terms, and find that average absolute election-level bias is about 2 percentage points, indicating that polls for a given election often share a common component of error. This shared error may stem from the fact that polling organizations often face similar di culties in reaching various subgroups of the population, and they rely on similar screening rules when estimating who will vote. Election-level bias accounts for much, but not all, the observed excess error; as a result, average election-level variance is also higher than implied by most reported margins of error. We conclude by discussing how these results help explain polling failures in the 2016 U.S. presidential election, and o↵er recommendations to improve polling practice.